A Bulk Data Transfer Bus

Storage, processing and transmission of very large data sets found in remotely sensed geographical information systems place heavy demands upon all aspects of a system design. The regularity of the data and the nature of the most common computational tasks allows advanced data tiling techniques to be used to good effect. A data transfer API which uses the notion of tiles as its fundamental data type allows each part of the system to describe both the physical aspects of parallel data management, and the computational and user level layouts. Mapping functions which describe the retitling of data from one tiling to another form the basic data transfer specification. Combinations of retilings can be specified before data is moved and a single, optimal remap performed that can be further optimised to best utilise the physical parameters of storage and communication devices. Bulk data is stored in a regular format for best performance, this requires that unstructured meta-data and control data be stored separately. A paradigm in which each data object is represented as a high level specification and a multi-dimensional array of simple data evolves. The system described can integrate with legacy applications through the provision of interfaces that create the illusion of files. These files are automatically generated upon demand and can have their content optimised so that the legacy application only sees exactly the data it needs. The Bulk Data Transfer Software Bus underpins the design of the DISCWorld system.